{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Find Detection Mistakes\n",
    "Annotations mistakes create an artificial ceiling on the performance of your models. However, finding these mistakes by hand is at least as arduous as the original annotation work! Enter FiftyOne.\n",
    "\n",
    "In this tutorial, we explore how FiftyOne can be used to help you find mistakes in your object detection annotations. To detect mistakes in classification datasets, check out the recipe in the Classification task.\n",
    "\n",
    "We’ll cover the following concepts:\n",
    "\n",
    "- Computing insights into your dataset relating to possible label mistakes\n",
    "- Visualizing mistakes in the FiftyOne App\n",
    "\n",
    "\n",
    "In order to compute mistakenness, your dataset needs to have [two detections](https://docs.voxel51.com/user_guide/using_datasets.html#object-detection) fields, one with your ground truth annotations and one with your model predictions.\n",
    "\n",
    "In this example, we’ll load the [quickstart](https://docs.voxel51.com/user_guide/dataset_zoo/datasets.html#dataset-zoo-quickstart) dataset from the FiftyOne Dataset Zoo, which has ground truth annotations and predictions from a PyTorch Faster-RCNN model for a few samples from the COCO dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fiftyone as fo\n",
    "import fiftyone.zoo as foz\n",
    "\n",
    "dataset = foz.load_zoo_dataset(\"quickstart\")\n",
    "\n",
    "session = fo.launch_app(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute Mistakeness\n",
    "Now we’re ready to assess the mistakenness of the ground truth detections. Mistakeness\n",
    "\n",
    "We can do so by running the [compute_mistakenness()](https://docs.voxel51.com/api/fiftyone.brain.html#fiftyone.brain.compute_mistakenness) method from the FiftyOne Brain:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fiftyone.brain as fob\n",
    "\n",
    "# Compute mistakenness of annotations in `ground_truth` field using\n",
    "# predictions from `predictions` field as point of reference\n",
    "fob.compute_mistakenness(dataset, \"predictions\", label_field=\"ground_truth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above method populates a number of fields on the samples of our dataset as well as the ground truth and predicted objects:\n",
    "\n",
    "New ground truth object attributes (in ground_truth field):\n",
    "\n",
    "- mistakenness (float): A measure of the likelihood that a ground truth object’s label is incorrect\n",
    "- mistakenness_loc: A measure of the likelihood that a ground truth object’s localization (bounding box) is inaccurate\n",
    "- possible_spurious: Ground truth objects that were not matched with a predicted object and are deemed to be likely spurious annotations will have this attribute set to True\n",
    "\n",
    "New predicted object attributes (in predictions field):\n",
    "\n",
    "- possible_missing: If a highly confident prediction with no matching ground truth object is encountered, this attribute is set to True to indicate that it is a likely missing ground truth annotation\n",
    "\n",
    "Sample-level fields:\n",
    "\n",
    "- mistakenness: The maximum mistakenness of the ground truth objects in each sample\n",
    "- possible_spurious: The number of possible spurious ground truth objects in each sample\n",
    "- possible_missing: The number of possible missing ground truth objects in each sample\n",
    "\n",
    "## Analyzing Results\n",
    "Let’s use FiftyOne to investigate the results.\n",
    "\n",
    "First, let’s show the samples with the most likely annotation mistakes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fiftyone import ViewField as F\n",
    "\n",
    "# Sort by likelihood of mistake (most likely first)\n",
    "mistake_view = dataset.sort_by(\"mistakenness\", reverse=True)\n",
    "\n",
    "# Print some information about the view\n",
    "print(mistake_view)\n",
    "\n",
    "# Show the samples we processed in rank order by the mistakenness\n",
    "session.view = mistake_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another useful query is to find all objects that have a high mistakenness, let's say > 0.95:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fiftyone import ViewField as F\n",
    "\n",
    "session.view = dataset.filter_labels(\"ground_truth\", F(\"mistakenness\") > 0.95)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking through the results, we can see that many of these images have a bunch of predictions which actually look like they are correct, but no ground truth annotations. This is a common mistake in object detection datasets, where the annotator may have missed some objects in the image. On the other hand, there are some detections which are mislabeled and are just wrong.\n",
    "\n",
    "In the example we can see two potential issues. One is the person on the TV is unlabeled. Two is the books on the bookcase are ambiguously labeled. Is one book a book, or is a group of books \"book\". FiftyOne helps you sort out issues exactly like this.\n",
    "\n",
    "![mistake-sample](assets/mistake-sample.webp)\n",
    "\n",
    "We can use a similar workflow to look at objects that may be localized poorly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = dataset.filter_labels(\"ground_truth\", F(\"mistakenness_loc\") > 0.85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![mistake-loc](https://cdn.voxel51.com/mistake-loc.webp)\n",
    "In some of these examples, there is not necessarily highly mistaken localization, there are just a bunch of small, relatively overlapping objects. In other examples, such as above, the localization is clearly off.\n",
    "\n",
    "The possible_missing field can also be useful to sort by to find instances of incorrect annotations.\n",
    "\n",
    "Similarly, possible_spurious can be used to find objects that the model detected that may have been missed by annotators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = dataset.match(F(\"possible_missing\") > 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we can find more of those pesky mistakes! Look at all the missing person annotations here!\n",
    "![mistake-missing](https://cdn.voxel51.com/mistake-missing.webp)\n",
    "\n",
    "In FiftyOne, we can tag our samples and export them for an annotation job with one of our labeling integrations: [CVAT](https://docs.voxel51.com/integrations/cvat.html), [Label Studio](https://docs.voxel51.com/integrations/labelstudio.html), [V7](https://docs.voxel51.com/integrations/v7.html), or [LabelBox](https://docs.voxel51.com/integrations/labelbox.html)! This can get our dataset back into tip-top shape to train again!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "You've learned to find annotation mistakes using `fob.compute_mistakenness()` and analyze results with mistakenness scores, localization errors, and missing/spurious detection fields.\n",
    "\n",
    "Next up: **Step 4: Use labeling integrations to fix mistakes and improve your dataset quality**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "OSS310",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
